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"I had hoped to find research to support or to conclusively oppose my belief that quality 
integrated education is the most promising approach. For every study that contains a 
recommendation, there is another, equally well documented study, challenging the 
conclusions of the first.. .No one seems to agree with anyone else's approach. But more 
distressing: no one seems to know what works." 

Senator Fritz Mondale's quote illustrates a common plight. Educational research often 
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produces contradictory results. Differences among studies in treatments, settings, 
measurement instruments, and research methods make research findings difficult to 
compare. Even frequent replications can prove inconclusive. Literature on a topic may 
be so extensive as to obscure trends with an overwhelming amount of information. 

Meta-analysis is a collection of systematic techniques for resolving apparent 
contradictions in research findings. Meta-analysts translate results from different studies 
to a common metric and statistically explore relations between study characteristics and 
findings. 

This digest first describes meta-analysis as a research method. The need and general 
approach are discussed. We then identify some common approaches toward 
conducting meta-analysis in education and outline their advantages and disadvantages. 

META-ANALYSIS AS A RESEARCH METHOD 

Gene Glass first used the term "meta-analysis" in 1976 to refer to a philosophy, not a 
statistical technique. Glass argued that literature review should be as systematic as 
primary research and should interpret the results of individual studies in the context of 
distributions of findings, partially determined by study characteristics and partially 
random. Since that time, meta-analysis has become a widely accepted research tool, 
encompassing a family of procedures used in a variety of disciplines. A recent search of 
the ERIC database identified over 800 articles written after 1980 that use or discuss 
meta-analysis. 

Meta-analysis responds to several problems in educational research. First, important 
issues are studied by numerous investigators. The amount of information on a given 
topic therefore is often overwhelming and not amenable to summary. Even when there 
are relatively few studies on a given topic, it is difficult to determine if outcome 
differences are attributable to chance, to methodological inadequacies, or to systematic 
differences in study characteristics. Informal methods of narrative review permit biases 
to remain easily undetected. Reviewers' biases can influence decisions about study 
inclusion, relative weights given to different findings, and analysis of relations between 
study features and outcomes. These biases can have clandestine effects when 
reviewers do not systematically seek to reduce them or provide sufficient information for 
readers to evaluate their extent. 

Meta-analysis typically follows the same steps as primary research. The meta-analyst 
first defines the review's purpose. Organizing frameworks can be practical or theoretical 
questions of varying scope, but they must be clear enough to guide study selection and 
data collection. Second, sample selection consists of applying specified procedures for 
locating studies that meet specified criteria for inclusion. Typically, meta-analyses are 
comprehensive reviews of the full population of relevant studies. Third, data are 
collected from studies in two ways. Study features are coded according to the objectives 
of the review and as checks on threats to validity. Study outcomes are transformed to a 
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connnnon metric so that they can be compared. A typical metric in educational research 
is the effect size, the standardized difference between treatment and control group 
means. Finally, statistical procedures are used to investigate relations among study 
characteristics and findings. 

Criticisms of meta-analysis tend to fall into two categories. Some complain that 
meta-analysis obscures important qualitative information by "averaging" simple 
numerical representations across studies. Other critics argue that research is best 
reviewed by a reflective expert who can sift kernels of insight from the confusing 
argumentation of a field. 

META-ANALYTIC APPROACHES 



VOTE-COUNTING - Some reviews categorize findings as significantly positive 
(favoring the treatment group), significantly negative, or nonsignificant. The category 
with the most entries is considered the best representation of research in this area. This 
as an inexact approach to integrating research. Vote-counting confuses treatment effect 
and sample size because statistical significance is a function of both. Given the modest 
power of typical educational research to detect true effects as statistically significant, 
conclusions from vote-counting can be very misleading. 

CLASSIC OR GLASSIAN META-ANALYSIS - Glass' early meta-analyses set the 
pattern for conventional meta-analysis: define questions to be examined, collect studies, 
code study features and outcomes, and analyze relations between study features and 
outcomes. These early meta-analyses, and later ones following this tradition, share 
three distinguishing features. First, "classic" meta-analysis applies liberal inclusion 
criteria. Glass argued that one should not disregard studies on the basis of study quality 
a priori; a meta-analysis itself can determine if study quality is related to variance in 
reported treatment effect. Second, the unit of analysis is the study finding. A single 
study can report many comparisons between groups and subgroups on different criteria. 
Effect sizes are calculated for each comparison. Third, meta-analysts using this 
approach may average effects from different dependent variables, even when these 
measure different constructs. 

Glassian meta-analysis has proven quite robust when submitted to critical re-analysis. 
Its use of conventional statistical tests render the method and its results accessible to 
most educational researchers. However, using study findings as the units of analysis 
produces nonindependent data and gives greater weight to studies with many 
comparisons. Averaging across constructs and including studies with obvious 
methodological flaws can confuse the reliability of findings. 

STUDY EFFECT META-ANALYSIS - Study effect meta-analysis alters the Glassian 
form in two ways. First, inclusion rules are more selective. Studies with serious 
methodological flaws are excluded. Second, the study is the unit of analysis. One effect 
size is computed for each study. This preserves the independence of the data and gives 
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equal weight to all included studies. Unfortunately, it also reduces the number of data 
points analyzed in the review. And, of course, a reviewer's biases may operate in 
decisions to exclude studies. 

TESTS OF HOMOGENEITY -- Some reviewers argue that conventional statistical tests 
are inappropriate for meta-analysis. Homogeneity tests were developed to determine 
the likelihood that variance among effect sizes is due only to sampling error. If the 
homogeneity statistic is significant for a group of studies, a procedure analogous to 
analysis of variance can be used. Studies are repeatedly divided into subgroups 
according to study features until within-group variation is nonsignificant. 

Numerous factors can cause variation in effect sizes: measurement unreliability, range 
restrictions, reporting errors, within-study statistical adjustments, unreported factors, etc. 
Homogeneity tests are very likely to indicate heterogeneity among effect sizes even 
when the variation is of no practical or theoretical importance. Successively dividing 
subgroups according to these tests can capitalize on chance and cause the incorrect 
identification of moderators. Kulik and Kulik defend conventional analysis of variance for 
meta-analysis and suggest that homogeneity tests may ignore an important nesting 
factor. 

PSYCHOMETRIC META-ANALYSIS - Hunter and Schmidt's approach to 
meta-analysis combines some of the best features of other approaches. All studies 
related to a given topic are gathered, regardless of quality. The distribution of effect 
sizes is corrected for sampling error, measurement error, range restriction, and other 
systematic artifacts. If the remaining variance is still large, effect sizes are grouped into 
subsets according to preselected study features, and each subset is meta-analyzed 
separately. Ideally, the meta-analysis should estimate true treatment effects under 
conditions typical of those represented in the studies and predict treatment effects 
under conditions determined by the reviewer. Unfortunately, this technique requires 
substantial information from individual studies for accurate correction of effect sizes. 
This information is not always available in research reports. 
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This publication was prepared with funding from the Office of Educational Research and 
Improvement, U.S. Department of Education under contract number RI88062003. The 
opinions expressed in this report do not necessarily re fleet the position or policies of 
OERI or the Department of Education. 
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